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REMARKS 

Claims 1 , 3-10 are rejected under 25 U.S.C. 103(a) as being 
unpatentable over Jian ct al (6,567,980) and l akemoto (6,335,742). The 
applicants rcspectflilly traverse this rejection on grounds that the Jain et al. 
reference does not teach or suggest selected points for which it is cited in Office 
Action of September 27, 2003. 

L,iClaim i: 

The following reviews various points made ia the Office Action of 
September 22, 2003 about what is shown in the Jain et al, reference and responds 
to each point: 

1, a GUI adanted to bro^e mciutes stored in a data hase (column 2 lin^s X 0-20, 
column 4 lines 20-35), including: a main level dispiav providinw links to other 
display levels (Fis^Jtl 

Jain ct ah does not describe at Fig. 2, a main level display providing links 
to other display levels. Instft;id, what is shown in Fig. 2 is described column 4 
lines 21-40 of the Jain et al, patejit. These lines state follows: 

FIG, 2 depicts an example user interface that is representative of 
the type of graphical user fnterfacf^ (GTU) than could he built 
around the Video Engine shown in FIG. 9. In FIG. 2, the Video 
Cataloger user interface is contained in a window 170. The main 
controls are exposed as menus and a tool bar 182, A panel 172 
displays the live video being digitized, with play, stop, etc, controls 
that interact remotely with the analog source via a deck controller 
240 (FIG, 3). Keyframes extracted during the capture process are 
di^Auyt^d in a panel 176, while the corresponding close-captian 
text and limecodes are displayed in a panel 178, A panel 184 
displays the user-defined clip annotations, created by marking in- 
and out-points. The columns J 86 and 188 display the in- and out- 
time codes for the marked clip, respectively, while the remaining 
columns 190, 192, 194 are an example of a user defined schema of 
labels to describe the clip. Finally, at the bottom of the window 
170 is a timeline 180 thuzt depicts the total time of the capture 
session, with a highlighted section corresponding to the currently 
selected range of keyframes. 

There is no teaching in the cited section or in that Fig. 2 comprises a main level 
displays that mformation providing links to other display levels. In fact, it 
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appears that main level shown in Fig. 2 is configured to provide all at once access 
to information regarding video clips, and selected key firanmes. 

2, a picture content iconic region having icons representine pictures according 
to predefined content catesories and picture metadata Fiss. 14, 17 column 14 
lines 27-65): 

According to the Jain et al. reference, Fig. 14 is "a fl owchart of the feature 
extraction process shown in FIG- 13" Figs. 13 and 14 are described at culujiiuj* U 
lines 52-65 and and column 1 2 lines ] -57. These lines state as follows: 

FIG, 13 details the metadata capture process 776 which is an 
important activity of the Video Engine 440 of FIG. 9, Uie metadata 
capture process 776 was first introduced in FIG, 12. 

The capture process 776 begins with the scheduling of a system 
timer event in step 804 set to go ojf [fraction (]/30)J of a second in 
the future. The control flow of the process 776 immediately 
proceeds to the Get Event step 806 where other .^stf>m p.vents, 
(besides the timer event) may be processed. When an event occurs, 
control passes to the Event Dispatcher 808 which decides if the 
event is one of the two types of es^ents: a normal GUI event, or the 
scheduled timer event. 

For a GUI evsnt, the event is first inspected in step 812 to 
deter minii if it in an End Gapture event, in which case the capture 
process loop terminates. If not, processing proceeds to step 816 to 
handle the GUI a\*cnt (such as keystroke., windouf resized, etc.). 
Some GUI events may generate metadata (if the user marked a 
video clip), which is determined instep 818, If metadata (a video 
clip) Vh'as in Jdct generated, that metadata is committed to the 
Metadata Track Index Manager 530 (FIG, 9) during step 820. This 
also necessitates a GUI redraw, so the affected parts of the GUI 
are marked for Redraw in step 822. 

If the event dispatched in 808 is the timer event., this signifies that 
feature extraction of metadata from the video signals fs to take 
place at a feature extraction process 810. Tftc details of the feature 
extraction process SlOare provided in con/unction with FIG. 14, 
Once feature extraction is complete, control moves to step 804 
where the next timer event is scheduled. 

This flow of activity is tied to the event model of the operating 
system under which the software application is running. The flow 
that is shown is an event model that is typical of a Windows MFC- 
based application. Other operating system platforms, such as 
Unix, have event models that differ somewhat. The event model 
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illustrates how the feature extraction process fits into an 
application event framework Note that, in the depicted 
embodiment, the (Jet Invent task 806 is a call out to Windows MFC, 
which processes Redraw Events by calling the Redraw method of 
the appropriate GUI elenmnis dirt^t^ily (tfiis process diagram does 
not ^*call" the Redraw methods directly). Note tftat it is acceptable 
if feature extraction takes more than [fraction {J/30)J sacond. 
14, Feature Extraction—Flowchart 

FIG. 14 details the feature extraction process 810, which is an 
important aspect of the present invention, relying on the innovative 
architecture of FIG, 9. 

The feature extraction process 810 begins at a start step 842 and 
proceeds to step 844 where the current time code is obtained by 
module 502 of FIG 9. litis time code is us^d by all feature 
extractors to time-stamp the metadata they extract. Next, all digital 
mpAin iM captured in step 846 by modules 504, 506. and 508 of 
FIG P. This digital media is then passed on to the Feature 
Extractor Framework 510 (FIG. 9) for processing. The Feature 
Extractor Framework 510 spccwns a process thread 850 for each 
feature extractor. Each feature extractor processes the digital 
media in step 852 in whatever -way it desires, for example^ i^xtruct 
a keyframe, classify the audio signal, etc. In certain cases, but not 
all, some metadata >vilt be generated from this process. Step 854 
determines if this is the case, and if so, the metadata is passed to 
the Metadata Track Index Manager 530(FJG. 9) during step 856. 
Since metadata is usually displayed in real-time in the GUI, the 
GUI is marked for redraw in step 858, One particular exemplary 
feature: extractor for video keyframes is described in the pending 
U.S. patent application entitled "Key Frame Selection" filed on 
Jun, 6, 1997. 

Wlxen all feature extractor threads complete, as determined at w^aii 
(synchronization) step 862, control is returned to the capture 
metadata process at end step 864. 

What appears Lo be dcsciibed heiebi the steps of cxtiactiiig metadata and features 
from the video content. As is stated at columns 12 lines 50-52 "since metadata is 
usually displayed in real-time in the GUI, the GUI is marked for redraw in step 
858.^* Thus, at the conclusion of the processes described in Fig. 13 and 14, what 
is disclosed in the Jain et aL reference is amending the displayed listing of 
metadata. There is no discussion in these lines of providing a content iconic 
region having icons representing pictures according to predefined content 
categories and picture metadata. It is also noted that there is no discussion of 
moving to another display level instead of the current level of the metadata is 
updated. 
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Fig. 1 7 is described as **an exemplaiy screen display seen as an output of 
the HTML output filter process of FIG. 1 6 while xtsing a client browser for the 
fnultimedia cataloger system shown in FIG. 1.'" As described at columns 12 lines 
60-65 and and one columns 1 J lines 1-32 Fig. 1 7 shows the following: 

The Output Filter Manager 560 (FJG. 8) may utilize a HTML 
output filter 564 in one embodiment Referring to FIG. 15, 
elements of FfGS, L 2 and 9 are ahown together as utilized in 
generating HTML output. The user may invoke a GUI command 
such as the "Save-As" command on th& "FiW menu 553, which in 
turn provides a list of output filter choices (HTML, Real Networks 
SMrr., XML. custom! etc.). When the HTML filter 564 is invoked, it 
accesses the metadata in the Metadata Track Index Manager 530 
and processes it into HTML form in a browser window 91 6 (FIG, 
1 7), which also involves keyframe images in a keyfratnejrame 1 76 
(FIG. 2) or 904 (FIG. 17), and the digital video 142 (FIG. 1) or as 
seen in a video frame B96 (FIG. 2 7), For insiunct^, hyperlinks may 
be formed from displayed keyframes to video sequences. The 
digital video 142 may or may not be s(sr\t^d by a content server 
140. For instance, it could be a simple file on the file system of the 
client computer or. say, a networked mass storage device visible to 
the computer. 

Some key features of the Video Cataloger HTML output are: 

a. The HTML files used to generate the display in ihfj bro^vstir 
window 916 (FIG. 1 7) are completely stand-alone, internally 
linked HTML, such that no Web sorter is rGquired. Exemplary 
HTML files are provided in the Appendix and are described in 
conjunction with FIG. 17 below. 

b. It incorporates play-back of digital video 142 from a file or from 
a video server 140. That is, the digital video may be streamed 
directly to the browser, or it may simply be played from a local file 
on disk. The stand- alone aspect is strengthened when //*e digital 
video is a local file. This way, all of the content (HTML, keyframes, 
digital y^ideo) could be paclzagcd up, compressed, and a-mailed to 
someone. 

c. All metadata is cross-referenced/cross-linked based on time- 
codes, 

d. Digital video is independent of the HTML representation— any 
digital video source can be linked into the playback frame. 

Thus, as ia shown in tbc3c litica, the window provided in Fig. 1 7 is a completely 
Stand alone HTML browser window with video clips linked to the HTML browser 
window such that the video clips can be accessed without need to access a web 
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server. There is no discussion in this section however, that Fig- 17 has a picture 
content iconin region having icons representing pictures according to predefined 
content categories and picttire metadata- Instead, Fig. 17 is merely an HTML 
version of what is shown in Fig. 2, which as has been discussed above. 

As noted in the Office Action, Fig. 17 is further described at column 14, 
lines 27-65, these lines state as follows: 

77. Example HTML Output-Screen Shot 

Referring to FIGS, 16 and 17, a screen shot of the HTML output as 
seen at a client bro'wser and as generated by the HTML output 
process 890 (tlG. lb) will be described. Element 896 corresponds 
to the video frame in the upper left portion of the screen display. 
Element 904 vurrti:>punds to the hsyfrauie frame in the loyver left 
portion of the screen display. Element 908 corresponds to the cc- 
text frame in the low>er right portion of iJip, screen display. Element 
912 corresponds to the clip frame in the upper right portion of the 
screen display. Element 916 corresponds to the whole browser 
window. As with most browsers, including Microsoft Explorer and 
Netscape Navigator, if the displc^ahle page is larger than the 
physical display, the browser will cause the page to be scrolled. 
Video data is retrieved by sending a time code to the embedded 
pluyi^r upplit:iition. The player application then retHes^ the video, 
seeks to the requested time code (in-time), and begins playback. 
The iisar can interrupt the plnyhnr.k using standard VCR type 
controls on the player. 

The HTML code for an exemplary screen display is provided in the 
Appendix. Sheets A. B, C A G, HI, Nand P are hereby 
incorporated by rejerence in their entirety and can be found in the 
USPTO file for application Ser. No. 09/134,499. Sheet A of the 
Appendix lists the directory names (clip and icons) and file names 
at a top level Sheet B lists the files in the clip directory, while 
sheets C D and E list the files in the icons directory Shet^t F lists 
the HTML code for the top level indexMml file which provides the 
framework for the display shown in the browser window 916 (FIG. 
1 7). Sheet G lists the contents of the topr.html file (as would be 
seen in the clip frame 912 (FIG. 1 7)). Sheet H lists the contents of 
the video^labeLhtml file. Sheet I lists the contents of the 
video_mbase,html file. Sheet J lists the contents of the 
video_netshow.html file. Shwt K lists iha contents of the 
video _jioproxy.html file. Sheet L lists the contents of the 
^dd^o_p^?s,html file. Sheat M lists the contents of the 
video_real.html file. Sheets J, K, L, and M may be used to provide 
the proxy video to allow different video formats to be displayed in 
the video frame 896 (FIG. 1 7). Sheet N lists the contents, including 
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a set of keyframes and corresponding timecodes (as would be seen 
in the keyframe frame 904 (FIG. 17)). of the OOOLhtmlfile in the 
clips directory. Sheet P Usts the contents, including a st^t vf Icons 
in a closed-caption text frame (as \90uld be seen in the cc-text 
frame 908 (FIG. 17)), of the OOOrJttml file in the clips directory. 
The remaining sheets in the Appendix are alternate instances of the 
contents sho^^m in exemplary^ sheets N and P. Of rnurst>, other 
programming languages besides HTML code could be used to 
implement hyperlinked output conversion, 

HtTC aijtiiiJi, what is described is fonming an tlTML based version of the image 
shown in Fig, 2, and nothing is described that shows a a picture content iconic 
region having icons representing pictures according to predefined content 
categories and picture metadata as is suggested in the Office Action. 

3. a erai>hical brOMK^f^r remon havwjt and indicia of &ranhical hr nx^p.rs uttliTed 
by the GUI fFw. 17) and further having a nluralitv of disulav levels linked to 
the main display level via one or mare icons C^ v^ 17. volumn 13 fines I^33)j 
Fig. 17 has been discussed in greater detail above. As noted above, as 
described at columns 12 lines 60-65 and columns 13 h'nes 1-32 Fig. 17 shows the 
following: 

The Output Filter Manager 560 (FIG. 8} may utilize a HTML 
output filter 564 in one embodiment. Referring to FIG. 15, 
elements of FIGS. 1, 2 and 9 are shown together as utilized in 
generating HTML output. Tlie user may invoke a GUI command 
such as the "Save-As" command on the "File" menu 553, which in 
turn provides a list of output filter choices (HTML, Real Networks 
SMIL, XML, custom, etc.). When the HTML filter 564 is invoked, it 
aocGSScs the metadata in the Metadata Track Index Manager- 530 
and processes it into HTML form in a browser window 916 (FIG, 
17), which also involves keyframe images in a keyframe frame 176 
(FIG. 2) or 904 (FIG, 1 7), and the digital video 142 (FIG. 1) or as 
seen in a video frame 896 (FIG. 1 7). For instance, hyperlinks may 
be formed from displayed keyframes to video sequences. The 
digital video 142 may or may not be served by a content sender 
MO. For instance, it could be a ^implt^file on the file system of the 
client computer or, say, a networked mass storage device visible to 
the computen 

Some key features of the Video Cataloger HTML output are: 

a. The HTML files used to generate the display in the browser 
window 916 (FIG, 17) are completely stand-alone, internally 
linked HTML, such that no Web server is required. Exemplary 
HTML files are provided in the Append ijs; and iircs di^^crihtsd in 
conjunction with FIG, 1 7 below. 
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b. It incorporates play-back of digital video 142Jrom a file or from 
a video server 140. That is, the digital video may be atraami^U 
directly to the browser, or it may simply be played from a local file 
on disk. The siarul- alone aspect is strengthened when the digital 
video is a local file. This v^ay, all of the content (HTML, keyframes, 
digital video) could be packaged up, compresscA, and e-mailed to 
someone. 

a All metadata is cross -referenced/cross-linked based on time- 
codes. 

d. Digital video is independent of the HTML representation—any 
digital video source can be linked into (he playback frama. 

llms, as is shown in these Jines the window provided in Fig. 17 is a completely 
staod alone HTML browser window with video clips linked to the HTML browser 
window such that the video clips cari be accessed without need to access a web 
server. Notably, as is described in column 14, lines 27 - 40, the video clips are 
presented in a video clip area 906. Thus, merely selecting a video clip from one 
of the displayed video cHp3 simply causes the video clip to be presented in video 
clip area 904 of the same display level. Further, column 14, lines 47-5 1 
explicitly states that it provides HTML code "for an exemplary screen display as 
is provided in the appendix." There is no apparent discussion in this section of 
formiag more than one such screen display thus, Fig. 17 does not appear to stat\d 
for the proposition that Jain et al . descrihes a difiploy further having a plurality of 
display levels linked to the main display level filed one or more as is suggested in 
the Office Action. 

4. a picture eroumn^ iconic region indicating files containins pictures in a 
database (Fie. 1 7) for picture srouvins^ 

Jain et al. shows an area P04 of the HTML image of Fig. 17 wherein 
thxuKibnail images of so-called key frames extracted from video streams arc 
positioned. These images can be clicked upon to cause a video clip associated 
therewith to be presented in area 896. What is displayed in this region are key 
frames extracted from tlie video. There is simply no discussion of key frame area 
904 as cont-nining icons, that ?;iicb icons represente groups of pictures in a 
database, or that the icons are grouped in any particular manner such as being 
bused upuxx coixtent or Jiietadata. 
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Thus, Jain et al. does not appear to support any of the above described 
points made in the Oflfir^ Action of September 22, 2003. To reject claims in a 
patent application under 35 U.S.C.S. § 103, an examiner must show an unrebutted 
prima facie case of obviousness* Tn the absence uf a prupci' pi iAwa facie case of 
obviousness, an applicant who complies with the other statutory requirements is 
entitled to a patent. As the case made in the Office Action of September 22, 2003 
relies upon Jain et ah to establish the pointe described above, and the points have 
now been rebutted, the applicants resectfliUy submit that claim 1, and all claims 
that depend upon claim 1 now stand ready for allowanco. 

Conclusion 

The prima facie case of obviousness made in the Office Action of 
September 22, 2003 stands rebutted Accordingly, all claims are in a condition 
for allowance, prompt notice of which is eamcjiilly »oUcited. 



Respectfully submitted, 




Roland R. Schindler II 
Attorney for AppUcant(s) 
Registration No. 40,802 
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